Personalized learning system and method for the automated generation of structured learning assets based on user data

ABSTRACT

Learning systems and methods of the present disclosure include generating a text document based on a digital file, tokenizing the text document, generating a semantic model based on the tokenized text document using an unsupervised machine learning algorithm, assigning a plurality of passage scores to a corresponding plurality of passages of the tokenized text document, selecting one or more candidate knowledge items from the tokenized text document based on the plurality of passage scores, filtering the one or more candidate knowledge items based on user data, generating one or more structured learning assets based on the one or more filtered candidate knowledge items, generating an interaction based at least on the one or more structured learning assets, and transmitting the interaction to a user device. Each passage score is assigned based on a relationship between a corresponding passage and the semantic model.

TECHNICAL FIELD

One or more embodiments of the invention relate generally to learningsystems and more particularly, for example, personalized learningsystems that automate the generation of structured learning assets basedon user data.

BACKGROUND

Electronic learning technologies are commonly used to help studentslearn, develop skills, and enhance their understanding of certainsubjects. For example, electronic learning technologies may provide aconvenient way to take a course online. However, these technologiesoften provide the same curriculum for many or all the students in thecourse. For example, a given course may have a common starting point anda common ending point for the students, regardless of each student'sweaknesses, strengths, and/or cognitive learning abilities.

Yet, the students typically vary in the way they learn, how quickly theylearn, and how they retain what they learn. As a result, the general“one-size-fits-all” approach provided to the students is oftenineffective, inefficient, and/or cumbersome to the students. Studentsusing the “one-size-fits-all” approach may apportion too much timereviewing subject matter that they know well at the expense of spendinginsufficient time reviewing subject matter that they know poorly. Inaddition, many students may be burdened with trying to identify theirown weaknesses and strengths for a given subject matter. Further,students may put forth efforts in determining how to apportion theirtime effectively to learn the subject matter. As a result, the studentsmay struggle with such burdens, they may not perform well on exams, andthey may become discouraged, leading to attrition which is one of themost significant problems in education today.

Electronic learning technologies also commonly lack access to a broadselection of source material. For example, a given online course may belimited to the contents of a single (or several) textbooks selected forthe course. Further, the online course may be limited to a subset ofchapters in the textbook, such as chapters selected by an instructor. Inanother example, an exam preparatory course may be limited to thecontent owned by the provider of the course. As a result, the studentsmay be confined to source material narrowly tailored to one (or several)out of many possible strategies for learning the subject matter. Yet, asnoted, students typically vary in the way they learn, how quickly theylearn, and how they retain what they learn. Thus, the limited scope ofavailable source material may result in restricting the student'slearning process.

SUMMARY

According to some embodiments, a learning system may include anon-transitory memory and one or more hardware processors configured toread instructions from the non-transitory memory to cause the system toperform operations. In some embodiments, the operations may includegenerating a text document based on a digital file, tokenizing the textdocument, generating a semantic model based on the tokenized textdocument using an unsupervised machine learning algorithm, assigning aplurality of passage scores to a corresponding plurality of passages ofthe tokenized text document, selecting one or more candidate knowledgeitems from the tokenized text document based on the plurality of passagescores, filtering the one or more candidate knowledge items based onuser data, generating one or more structured learning assets based onthe one or more filtered candidate knowledge items, generating aninteraction based at least on the one or more structured learningassets, and transmitting the interaction to a user device. Each passagescore may be assigned based on a relationship between a correspondingpassage and the semantic model.

According to some embodiments, a method may include extracting text fromdigital content, performing semantic analysis of the text to generate asemantic model, scoring each passage of the text based on a relationshipbetween the passage and the semantic model, selecting one or morecandidate knowledge items from the text based on the scoring, filteringthe one or more candidate knowledge items based on user data, andgenerating a structured learning asset based on the one or more filteredcandidate knowledge items.

According to some embodiments, a non-transitory machine-readable mediummay have stored thereon machine-readable instructions executable tocause a machine to perform operations. In some examples, the operationsmay include generating a text document based on received digitalcontent, tokenizing the text document, performing semantic analysis ofthe tokenized text document to generate a semantic model, scoring eachpassage of the tokenized text document based on a relationship betweenthe passage and the semantic model, selecting one or more candidateknowledge items from the tokenized text document based on the scoring,filtering the one or more candidate knowledge items based on user data,and generating a structured learning asset based on the one or morefiltered candidate knowledge items.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are simplified diagrams of a learning system accordingto some embodiments.

FIGS. 2A-2C are simplified diagrams of a learning system and aninstructor device according to some embodiments.

FIGS. 3A-3C are simplified diagrams of processes that may be performedby learning systems and/or in accordance with an embodiment of thedisclosure.

FIGS. 4A-4B are simplified diagrams of a method for extracting knowledgeitems from digital content according to some embodiments.

FIG. 5 is a simplified diagram of templates for structured learningassets and user devices in accordance with an embodiment of thedisclosure.

FIG. 6A is a simplified diagram of a learning system with a natural userinterface in a cloud-based configuration in accordance with anembodiment of the disclosure.

FIG. 6B is a simplified diagram of a learning system with a natural userinterface in a device-based configuration in accordance with anembodiment of the disclosure.

FIG. 7 is a simplified diagram of a method for providing a naturalinteraction with a learning system according to some embodiments.

DETAILED DESCRIPTION

FIGS. 1A and 1B are simplified diagrams of a learning system 100according to some embodiments. FIG. 1A illustrates a block diagram oflearning system 100 including content editor 102, knowledge item bank104, adaptive engine 106, and user devices 108. In one embodiment,learning system 100 may be implemented with a variety of electroniclearning technologies. For example, learning system 100 may beimplemented with web and/or mobile online courses, exam preparatorycourses, and foundational courses involving large amounts of materialsor contents, such as courses teaching medicine, dentistry, law,engineering, aviation, and/or other disciplines. Yet, learning system100 may be implemented through kindergarten courses, elementary schoolcourses, high school courses, and also through college courses as well.Yet further, learning system 100 may be implemented with institutionaltraining courses (e.g., training courses used by businesses,corporations, governments, and/or military entities) and/or professionaltraining courses, such as courses to obtain professional certifications(e.g., emergency technician certifications).

In one embodiment, learning system 100 may be implemented to improvevarious electronic learning technologies. For example, learning system100 may improve the technologies to adapt to each student's weaknesses,strengths, and/or cognitive learning abilities. In particular, learningsystem 100 may generate individualized sessions for each student tostudy materials over time, thereby building long-term retention asopposed to cramming to provide short-term retention followed by a lossof the retention. Learning system 100 may also effectively optimize eachstudent's time spent studying based on the student's learningprogressions. For example, learning system 100 may determine when eachstudent is apt to learn and retain information. For example, learningsystem 100 may determine a student is apt to learn in the morning versusin the afternoon, among other possibilities.

In another embodiment, learning system 100 may improve electroniclearning technologies by resolving challenges described herein regardingcourses being limited to certain contents selected for the course.Learning system 100 may have the capability to integrate with a growinglibrary of digital source materials including, for example, multipletext books, a collection of portable document formats (PDFs), contentimages, multimedia videos, audio content, and/or other resources with acombination of the materials described herein. For example, learningsystem 100 may be used with one hundred text books from a firstpublisher, fifty text books from a second publisher, twenty textbooksfrom a third publisher, and thirty text books from a fourth publisher,among other contents from a combination of multiple publishers. In oneexample, learning system 100 may be capable of integrating withelectronic reader applications to provide the personalized learningprocesses in numerous types of mobile electronic devices, includingtablet devices, electronic reader devices, and/or personal computingdevices.

As further described herein, content editor 102 may be a content editorprocessor in wired or wireless communication with user devices 108,possibly including instructor and/or learner devices. In particular,content editor 102 may be in communication with a network (e.g., a basestation network) that is also in wireless communication with userdevices 108. Such wireless communication may be implemented inaccordance with various wireless technologies including, for example,code division multiple access (CDMA), Long Term Evolution (LTE), GlobalSystem for Mobile Communications (GSM™), or other standardized orproprietary wireless communication techniques. In another example, thewireless communication may be implemented with short-range wirelesstechnologies, such as Wi-Fi™. Bluetooth™, or other standardized orproprietary wireless communication techniques.

Content editor 102 may be implemented to receive, retrieve, and processcontent 112 from user devices 108, such as learner devices and/orinstructor devices. Content 112 may be a content data packet thatincludes texts from digital source materials, possibly including textsfrom electronic textbooks, where the texts may be highlighted by one ormore users. In one embodiment, highlighted materials may include markeddigital source materials, such as underlined, bolded, and/or italicizedtext or content, among other markings discussed further herein. In oneexample, content 112 may include figures, images, videos, and/or audiocontents. In one embodiment, content editor 102 may identify andtransmit a set of knowledge items 114 based on content 112. Knowledgeitems 114 may be objects and/or the building blocks of the learningprocesses as further described herein. Content editor 102 may transferknowledge items 114 to knowledge item bank 104 to store knowledge items114.

Adaptive engine 106 may retrieve knowledge items 116 from knowledge itembank 104. Adaptive engine 106 may also be in wired or wirelesscommunication with user devices 108, possibly including instructorand/or learner devices. In particular, adaptive engine 106 may be incommunication with a network (e.g., a base station network) that is alsoin wireless communication with user devices 108. Such wirelesscommunication may be implemented in accordance with various wirelesstechnologies including, for example, CDMA, LTE. GSM™, or otherstandardized or proprietary wireless communication techniques. Inanother example, the wireless communication may be implemented withshort-range wireless technologies, such as Wi-Fi™, Bluetooth™, or otherstandardized or proprietary wireless communication techniques.

Adaptive engine 106 may create and transmit interactions 118 to userdevices 108. In one embodiment, adaptive engine 106 may generateinteractions 118 based on knowledge items 116 and transmit interactions118 to user devices 108 for the learners to respond. In one example,adaptive engine 106 may determine the modality of interactions 118, suchas a multiple choice question, a matching question, and/or afill-in-the-blank question. In another example, adaptive engine 106 maydetermine a schedule to identify when to transmit interactions 118 touser devices 108 for the learners to respond. In particular, adaptiveengine 106 may determine when a learner is apt to learn and retaininformation. In one example, adaptive engine 106 may transmitinteractions 118 during learning sessions (e.g., intra trial) and/orbetween learning sessions (e.g., inter trial).

In various embodiments, learning system 100 may operate a feedback loopwith content editor 102, knowledge item bank 104, adaptive engine 106,and user devices 108. In one embodiment, user devices 108 may transmitcontent 112 to content editor 102. Content editor 102 may generate andtransmit knowledge items 114 based on content 112. Knowledge item bank104 may store knowledge items 114. Adaptive engine 106 may generate andtransmit interactions 118 based on stored items 116, and the process maycontinue accordingly. In one example, adaptive engine 106 may determinewhich interactions 118 to generate and when to transmit interactions 118to user devices 108 based on content 112 received from user devices 108.

FIG. 1B illustrates a block diagram of learning system 100 furtherincluding interaction applications 109, content analytics data 110, andlearner analytics data 111 in accordance with an embodiment of thedisclosure. FIG. 1B further illustrates content editor 102, knowledgeitem bank 104, and adaptive engine 106 as further described herein.

In one embodiment, each of user devices 108 may have installed arespective interaction application 109. Respective interactionapplications 109 may be displayed on user devices 108 and respectiveinteractions 118 may be provided by adaptive engine 106. Based onrespective interactions 118 provided, respective user inputs 120 may beprovided with each interaction application 109. For example, based onrespective user inputs 120, respective responses 122 may be generatedand transmitted to adaptive engine 106. In one embodiment, there may bea continuous cycle with adaptive engine 106, interactions 118, andresponses 122 from user devices 108 driven by the learning processeswith respective interaction applications 109.

In one embodiment, adaptive engine 106 may generate and transmitrespective learner analytics data 111 to each device of user devices108. Respective learner analytics data 111 may inform each learnerregarding the learner's performance and/or performance results based onrespective responses 122 to respective interactions 118. In one example,learner analytics data 111 may be transmitted to user device 108, e.g.,an instructor device, to inform the instructor regarding the learners'performances, group performances, and/or class performances andprogressions, among other indicators of one or more classes. In oneembodiment, the instructor may be an educator, a teacher, a lecturer, aprofessor, a tutor, a trainer, and/or a manager, among other possibleindividuals.

In one embodiment, adaptive engine 106 may generate content analyticsdata 110 based on the respective responses 122 from each interactionapplication 109 of user devices 108. Content analytics data 110 mayindicate performance results based on the respective responses 122. Inparticular, content analytics data 110 may indicate how the learners areperforming, whether the learners are retaining information associatedwith knowledge items 116, and/or whether the learners are progressingaccordingly. Content analytics data 110 may be transmitted to contenteditor 102. In one example, content editor 102 may generate additionalknowledge items 114 based on content analytics data 110.

In one embodiment, content analytics data 110 may inform contentcreators, publishers, and/or instructors regarding how the learnersperform based on responses 122. Content analytics data 110 may indicateknowledge items 116 that learners may understand well and also knowledgeitems 116 that may be challenging to learners. For example, contentanalytics data 110 may be used to generate a copy of digital sourcematerials, such as electronic textbooks, that illustrate knowledge items116 that may be challenging to learners. Such analytics data 110 mayimprove electronic learning technologies by providing challenging items116 in digital source materials, such as text books. In some example,learners are able to review digital source materials, such as textbooks, while also viewing challenging knowledge items 116 of thematerials.

FIGS. 2A-2C are simplified diagrams of a learning system 200 and aninstructor device 240 according to some embodiments. FIG. 2A illustratesa block diagram of learning system 200 including learner devices 204,206, and 208. The various components identified in learning system 100may be used to provide various features in learning system 200 in oneembodiment. In particular, content editor 202 may take the form ofcontent editor 102 as further described herein.

Learner device 204 may be a tablet device that displays knowledge items220 and 222. Knowledge item 220 may provide. “Photosynthesis is nothighly efficient, largely due to a process called photorespiration.”Knowledge item 222 may provide, “Cr and CAM plants, however, have carbonfixation pathways that minimize photo respiration.” In one embodiment,learner device 204 may include an interaction application, such asinteraction application 109, that displays and highlights knowledgeitems 220 and 222, among other contents. For example, a learner mayhighlight knowledge items 220 and 222 with the interaction application.Learner device 204 may generate and transmit content 214 to contenteditor 202. For example, content 214 may be a content data packet thatincludes knowledge items 220 and 222. As a result, content editor 202may identify knowledge items 220 and 222 from digital source materialsas further described herein.

Learner device 206 may be a smartphone that displays knowledge item 220.Knowledge item 220 may provide, “Photosynthesis is not highly efficient,largely due to a process called photorespiration.” In one embodiment,learner device 206 may include an interaction application, such as, forexample, interaction application 109, that displays and highlightsknowledge item 220 among other content. For example, a learner mayhighlight knowledge item 220 with the interaction application. Learnerdevice 206 may generate and transmit content 216 to content editor 202.For example, content 216 may be a content data packet that includesknowledge item 220. As a result, content editor 202 may identifyknowledge item 220 from the digital source materials as furtherdescribed herein.

Learner device 208 may be a smartphone that displays knowledge item 224.Knowledge item 224 may provide. “A photosystem consists of chlorophyll,other pigments, and proteins.” In one embodiment, learner device 208 mayinclude an interaction application, such as, for example, interactionapplication 109, that displays and highlights knowledge item 224 amongother content. For example, a learner may highlight knowledge item 224with the interaction application. Learner device 208 may generate andtransmit digital content 218 to content editor 202. For example, content218 may be a content data packet that includes knowledge item 224. As aresult, content editor 202 may identify knowledge item 224 from thedigital source materials as further described herein.

FIG. 2B illustrates a block diagram of learning system 200 furtherincluding adaptive engine 226 in accordance with an embodiment of thedisclosure. The various components identified in learning system 100 maybe used to provide various features in learning system 200 in oneembodiment. For example, adaptive engine 226 may take the form ofadaptive engine 106 as further described herein.

Adaptive engine 226 may generate and transmit interaction 228 to learnerdevice 204. For example, interaction 228 may be generated based onknowledge items 220 and 222 received by learner device 204 andidentified by content editor 202 from the digital source materials asfurther described herein. In one embodiment, interaction 228 may be amultiple choice question that provides. “Which of the following is nothighly efficient, largely due to a process called photo-respiration? A.Photosynthesis, B. Photoautotrophs. C. Cyanobacteria, and D. Corneliusvan Niel.” As noted, learner device 204 may include an interactionapplication, such as, for example, interaction application 109, thatdisplays interaction 228. In one example, the interaction applicationmay receive a learner input that indicates response 234 including aselection of A. B. C, or D. For example, response 234 may include thecorrect answer with the selection of A. As a result, response 234 may betransmitted to adaptive engine 226. Adaptive engine 226 may generate andtransmit interaction 230 to learner device 206. Interaction 230 may begenerated based on knowledge item 220 received by learner device 206 andidentified by content editor 202 from the digital source materials asfurther described herein. In one embodiment, interaction 230 may be afill-in-the-blank question that provides, “Photosynthesis is not highlyefficient, largely due to a process called.” As noted, learner device206 may include an interaction application, such as, for example,interaction application 109, that displays interaction 230. In oneexample, the interaction application may receive a learner input thatindicates response 236. For example, response 236 may include thecorrect answer of “photo-respiration.” As a result, response 236 may betransmitted to adaptive engine 226.

Adaptive engine 226 may generate and transmit interaction 232 to learnerdevice 208. Interaction 232 may be generated based on knowledge item 224received by learner device 208 and identified by content editor 202 fromthe digital source materials as further described herein. In oneembodiment, interaction 232 may be a fill-in-the-blank question and/orinteraction that provides, “A photosystem consists of _(——————), otherpigments, and proteins.” As noted, learner device 208 may include aninteraction application, such as, for example, interaction application109, that displays interaction 232. In one example, the interactionapplication may receive a learner input that indicates response 238. Forexample, response 238 may include “chloroplast” instead of the correctanswer “chlorophyll” and may be transmitted to adaptive engine 226.

FIG. 2C illustrates instructor device 240 in accordance with anembodiment of the disclosure. The various components identified inlearning systems 100 and 200 may be used to provide various features ofinstructor device 240 in one embodiment. For example, instructor device240 may take the form of one user device 108. In one example, instructordevice 240 may display an electronic copy of digital source materials242. Knowledge item 220 displayed by learner devices 204 and 206 mayalso be displayed by instructor device 240. Knowledge item 222 displayedby learner device 204 may also be displayed by instructor device 240.Knowledge item 224 displayed by learner device 208 may also be displayedby instructor device 240.

In one embodiment, knowledge items 220, 222, and 224 may be displayedbased on content analytics data such as, for example, content analyticsdata 110 from adaptive engine 106. For example, content editor 102 maygenerate knowledge items 220, 222, and 224 for display on instructordevice 240 based on content analytics data 110.

Knowledge item 220 may be highlighted and displayed by instructor device240. For example, knowledge item 220 may be highlighted based onresponses 234 and 236 including correct answers of the selection A andthe fill-in-the-blank “photorespiration.” respectively. In one example,knowledge item 220 may be highlighted and displayed by instructor device240 with a first color, such as, a green color that indicates thelearners' understanding of knowledge item 220.

Knowledge item 222 may also be displayed by instructor device 240. Forexample, knowledge item 222 may be displayed without highlights,possibly based on the learner's not having been tested on knowledge item222.

Knowledge item 224 may be highlighted and displayed by instructor device240. For example, knowledge item 224 may be highlighted based onresponse 234 including an incorrect answer “chloroplast” instead of thecorrect answer “chlorophyll”. In one example, knowledge item 224 may behighlighted with a second color, such as, a red color that indicates thelearner's understanding or lack of understanding of knowledge item 224.Knowledge items 220, 222, and 224, among other knowledge itemscontemplated in FIG. 2C, may provide an instructor with an indication oflearner weaknesses, strengths, and how to apportion class and studyingtime effectively. Such knowledge items, highlighted and/or nothighlighted, may improve electronic learning technologies by providingchallenging knowledge items, such as knowledge item 224, in digitalsource materials 242 in one or more highlights. In some example,instructors are able to review digital source materials 242, such astext books, while also viewing challenging knowledge items 224 indigital source materials 242 with one or more highlights.

In one example, knowledge items 220, 222, and 224, among other knowledgeitems, may be displayed and highlighted on learner device 204 based onresponse 234. In particular, knowledge item 220 may be highlighted ingreen based on response 234 and knowledge items 222 and 224 may not behighlighted since they may not have yet been tested. In another example,knowledge items 220, 222, and 224, among other knowledge items, may bedisplayed and highlighted on learner device 206 based on response 236.In particular, knowledge item 220 may be highlighted in green andknowledge items 222 and 224 may not be highlighted since they may nothave yet been tested. In another example, knowledge items 220, 222, and224, among other knowledge items, may be displayed and highlighted onlearner device 208 based on response 238. In particular, knowledge items220 and 222 may not be highlighted since they may not have yet beentested and knowledge item 224 may be highlighted in red based onincorrect response 238. As a result, learner devices 204, 206, and 208may provide the respective learners with an indication of each learner'sweaknesses, strengths, and how to apportion studying time effectively.

FIGS. 3A-3C illustrate processes 300, 330, and 350 that may be performedby learning systems 100 and/or 200 in accordance with an embodiment ofthe disclosure. Learning systems 100 and/or 200 may perform processes300, 330, and 350 without user actions, particularly without learnerand/or instructor actions.

Although various blocks of FIGS. 3A-3C are primarily described as beingperformed by one or more of learning systems 100 and/or 200, otherembodiments are also contemplated wherein the various steps may beperformed by any desired combination of learning systems and/or userdevices, such as learner devices and/or instructor devices describedherein.

At steps 302 a and 302 b, learning systems 100 and/or 200 may retrieveor obtain various types of digital source materials as described herein.In one example, the digital source material may include unstructuredcontent, as depicted in step 302 a. For example, the unstructuredcontent may include an audio and/or video representation of a learningexperience (e.g., a lecture, a tutorial, a demonstration, etc.), atextual and/or pictorial representation of learning material (e.g., atextbook, a webpage, an instruction manual, etc.), metadata, and/or thelike. The unstructured content may be unimodal (e.g., pure text) and/ormultimodal (e.g., combining text, audio, and/or images). In anotherexample, the digital source material may include structured content, asdepicted in step 302 b. For example, the structured content may includedatabase content, a knowledge graph, content that has been previouslyretrieved and processed by learning systems 100 and/or 200 (e.g.,content that was retrieved as unstructured content and subsequentlyprocessed into a structured format), and/or the like. In furtherexamples, the digital source material may include any suitablecombination of structured and/or unstructured content. In some examples,the digital source materials may include a plurality of source materialsretrieved one at a time and/or at the same time.

At steps 304 a and 304 b, learning systems 100 and/or 200 may extractand/or select a set of knowledge items from the digital source material.Each knowledge item generally corresponds to a discrete fact, figure,and/or illustration (and/or a group or related facts, figures, and/orillustrations) presented in the digital source material that the usermay desire to learn. In some examples, the knowledge items may beextracted from unstructured content, as depicted in step 304 a. In someexamples, extracting the knowledge items may include performing image,character, and/or speech recognition of image, textual, and/or audiodata included in the digital source material. Extracting the knowledgeitems may further include parsing, subsetting, sorting, classifying,clustering, and/or otherwise organizing the digital source material tosupport the identification of knowledge items. In some examples, theknowledge items may be selected from structured content, as depicted instep 304 b. In some examples, selecting the knowledge items may includefinding items in the structured content (e.g., traversing a knowledgegraph, searching a database, etc.) that the user may desire to learn.

At step 306, learning systems 100 and/or 200 may process the set ofknowledge items as further described herein. In one example, learningsystems 100 and/or 200 may identify, tag, label, categorize, format,deduplicate, score, sort, index, and/or store the knowledge items. Forexample, learning systems 100 and/or 200 may organize the knowledgeitems by subject and/or by topic.

At step 308, learning systems 100 and/or 200 may filter the set ofknowledge items based on user data and/or direct user feedback.Filtering based on direct user feedback may include asking the user torate, select, add, remove, and/or edit the knowledge items. Filteringbased on user data may include identifying one or more learningobjectives of the user. In some examples, the learning objectives maycorrespond to a query provided manually by the user that identifiessubject matter that the user desires to learn. In some examples, thelearning objectives may be generated automatically based on, forexample, the user's existing knowledge. For example, the learningobjectives may be automatically selected such that the user is not askedto re-learn a knowledge item that the user has already strongly learned.Learning systems 100 and/or 200 may compare the user's level ofknowledge to a predetermined threshold to determine whether re-learninga particular knowledge item is warranted. In some examples, the learningobjectives may be generated based on a combination of manually providedand automatically generated user data.

According to some embodiments, learning systems 100 and/or 200 mayfilter the set of knowledge items by determining a relevance of aparticular knowledge item to the learning objective. For example,learning systems 100 and/or 200 may calculate a relevance score for aparticular knowledge item and determine whether the relevance scoremeets a minimum threshold. When the particular knowledge item does notmeet the minimum threshold for relevance, that item is filtered out. Theminimum threshold may correspond to an absolute threshold and/or arelative threshold. For example, filtering using an absolute thresholdmay include selecting knowledge items with a relevance score greaterthan a predetermined value (and/or filtering out knowledge items with arelevance score less than a predetermined value), whereas filteringusing a relative threshold may include selecting the top N scores amongthe set of knowledge items (and/or filtering out the bottom N scores).

According to some embodiments, learning systems 100 and/or 200 mayfilter the set of knowledge items by determining a similarity of aparticular knowledge item to other knowledge items. For example,learning systems 100 and/or 200 may determine whether the particularknowledge item is a duplicate and/or a near duplicate of anotherknowledge item. When the particular knowledge item is a duplicate and/ora near duplicate, that item is filtered out.

At step 310, learning systems 100 and/or 200 may process the set offiltered knowledge items, where the filtered knowledge items correspondto those that are not filtered out at process 308. In one example,learning systems 100 and/or 200 may identify, index, and/or store theset of filtered knowledge items.

At step 312, learning systems 100 and/or 200 may generate a set ofstructured learning assets based on the set of filtered knowledge itemsas described herein. In one example, learning systems 100 and/or 200 mayconvert the filtered knowledge items to structured learning assets orresources. In some examples, learning systems 100 and/or 200 may convertthe filtered knowledge items by copying information associated with thefiltered knowledge items into a template corresponding to the structuredlearning asset. For example, the template may include a diagram fieldand a caption field. Accordingly, learning systems 100 and/or 200 mayinsert a diagram from the filtered knowledge items into the diagramfield and a corresponding caption for the diagram into the captionfield. In another example, learning systems 100 and/or 200 may determinea relationship between filtered knowledge items, and/or a relationshipbetween concepts within a given filtered knowledge item, to form aquestion and answer pair. For example, if the filtered knowledge itemcorresponds to “Albert Einstein was born in Germany,” the structuredlearning asset may correspond to “Question: Where was Albert Einsteinborn?” and “Answer: Germany.”

At step 314, learning systems 100 and/or 200 may process the set ofstructured learning assets. In one example, learning systems 100 and/or200 may identify, index, and/or store the set of structured learningassets or resources. In another example, learning systems 100 and/or 200may process the structured learning assets without user interactions(e.g., user highlights) as described herein.

At step 316, learning systems 100 and/or 200 may generate one or moreinteractions based on the structured learning assets. In one example, acloud-based personalized learning engine and/or artificial intelligenceengine, such as adaptive engines 106 and/or 226, may generate the one ormore interactions. In some examples, the one or more interactions may begenerated based on a user's history with similar structured learningassets. For example, learning systems 100 and/or 200 may access userfeedback from step 308 and/or user data from step 320 (as describedbelow) associated with a previously processed structured learning assetto generate the one or more interactions.

At step 318, learning systems 100 and/or 200 may generate a personalizedlearning experience based on the one or more interactions. In oneexample, learning systems 100 and/or 200 may generate the personalizedlearning experiences with interaction applications 109 described herein.

At step 320, learning systems 100 and/or 200 may generate user data thatmay be used at step 308 described herein to filter the set of knowledgeitems. In some examples, the user data may include one or more metricsthat indicate the user's level of retention and/or understanding of thedigital source material corresponding to the knowledge items. In someexamples, the one or more metrics may be determined based on the learnerresponses to the one or more interactions of the personalized learningexperience. For example, the one or more metrics may be determined basedon whether responses 234, 236, and/or 238 are correct or incorrect. Insome examples, the one or more metrics may further be determined basedon historical user responses associated with similar knowledge itemsand/or interactions that reflect the user's understanding of the currentknowledge items.

In some embodiments, the user data may include direct user feedback.Consistent with such embodiments, at step 322, learning systems 100and/or 200 may present the set of knowledge items extracted from thedigital source material at step 306 to the user via a user interface. Bypresenting the knowledge items to the user, the user has the opportunityto provide direct feedback to learning systems 100 and/or 200 regardingthe relevance of each of the knowledge items to the user's learningobjectives.

At step 324, learning systems 100 and/or 200 may receive direct userfeedback via the user interface that may be used at step 308 describedherein to filter the set of knowledge items. The direct user feedbackmay include ratings, selections, additions, removals, edits, and/or anyother type of user input that indicates the relevance of the knowledgeitems to the user's learning objectives.

Referring now to FIG. 3B, steps 332-344 of process 330 may relate tosteps 302-324 of process 300. In one example, one or more of steps302-324 may be sub-steps to process 330 and one or more steps 332-344may be sub-steps to process 300.

At step 332, learning systems 100 and/or 200 may retrieve or obtainvarious types of digital source materials as described herein. In theexample depicted in FIG. 3B, the digital source material includes awebpage on exoplanets.

At step 334, learning systems 100 and/or 200 may extract a set ofknowledge items from the digital source materials as described herein.Continuing the example depicted in FIG. 3B, in which the digital sourcematerial includes a webpage on exoplanets, the knowledge items mayinclude facts, figures, images, and/or multimedia clips related toexoplanets, as well as associated topics and/or sub-topics such asorbits, non-solar stars, detections, polarimetry, gravitationalmicrolensing, and/or transit photometry, among other possibilities.

At step 336, learning systems 100 and/or 200 may filter the set ofknowledge items based on user data as described herein. For example,when the user data indicates that the user has not yet learned aboutdetection methods for exoplanets, the set of filtered knowledge itemsmay include a knowledge item that lists detection methods forexoplanets. Notably, learning systems 100 and/or 200 may filter out aknowledge item that defines the term “exoplanet” when the user dataindicates that the user already knows what an exoplanet is.

At step 338, learning systems 100 and/or 200 may generate a set ofstructured learning assets that correspond to the set of filteredknowledge items. In one example, learning systems 100 and/or 200 maygenerate the structured learning assets based on templates as furtherdescribed herein. For example, learning systems 100 and/or 200 maygenerate a structured learning asset based on a “detection method”template that includes the following fields: a diagram of the detectionmethod, advantages of the detection method, and disadvantage of thedetection method. The fields of the “detection method” template may bepopulated using data from the filtered knowledge items, such as adiagram of transit photometry, an advantage of transit photometry, and adisadvantage of transit photometry.

At step 340, learning systems 100 and/or 200 may generate one or moreinteractions based on the structured learning assets. In one example, acloud-based personalized learning engine and/or artificial intelligenceengine, such as adaptive engines 106 and/or 226, may generate the one ormore interactions. For example, learning systems 100 and/or 200 maygenerate an interaction that includes a question. “Can you name oneadvantage of using Transit Photometry as an Exoplanet Detection Method?”The question may be a multiple choice question, a matching question,and/or a fill-in-the blank question, among other possible questions,formats, and/or modalities.

At step 342, learning systems 100 and/or 200 may generate a personalizedlearning experience based on the one or more interactions describedherein. In one example, learning systems 100 and/or 200 may generate thepersonalized learning experiences with interaction applications 109described herein.

At step 344, learning systems 100 and/or 200 may generate user data thatmay be used at step 336 described herein to filter the set of knowledgeitems. In one example, the user data may include or indicate the learnerhistory and/or learner responses to the one or more interactions of thepersonalized learning experience. For example, the user data mayindicate, “‘Detection Methods’ is not learned by current user; strongsemantic link to ‘Exoplanet’ in Learning AI” and/or “Exoplanetdefinition already learned strongly by current user.” As describedabove, based on the user data, ‘Detection Methods’ may be included inthe set of filtered knowledge items because it is not yet learned by theuser, but ‘Exoplanet definition’ may be filtered out because it isalready learned strongly.

Referring now to FIG. 3C, steps 352-364 of process 350 may relate tosteps 302-324 and/or 332-344 of processes 300 and/or 330. In oneexample, one or more of steps 302-324 and/or 332-344 may be sub-steps toprocess 350 and one or more steps 352-364 may be sub-steps to processes300 and/or 350.

At step 352, learning systems 100 and/or 200 may receive or obtainvarious types of digital source materials as described herein. In theexample depicted in FIG. 3B, the digital source material includes avideo on gravitational waves.

At step 354, learning systems 100 and/or 200 may extract a set ofknowledge items from the digital source materials described herein.Continuing the example depicted in FIG. 3B, in which the digital sourcematerial includes a video on gravitational waves, the knowledge itemsmay be extracted from audio and/or image analysis of the video fromvideo metadata, from a transcript of the video, and/or from tagged wordsfrom video excerpts. The knowledge items may include facts, figures,images, and/or multimedia clips related to gravitational waves, as wellas associated topics and/or sub-topics such as the cause ofgravitational waves, highly energetic astrophysical processes,historical figures such as Einstein, and the year when the waves wereproposed, among other possibilities.

At step 356, learning systems 100 and/or 200 may filter the set ofknowledge items based on user data as described herein. For example,when the user data indicates that the user is interested more in thehistory of astrophysics than the science of astrophysics, the set offiltered knowledge items may include items associated with theprediction of gravitational waves. As such, the filtered knowledge itemsmay include “Albert Einstein” (i.e., the scientist who predictedgravitational waves), “1916” (i.e., the year gravitational waves werepredicted), and/or “emerges from 1915 theory of general relativity”(i.e., the historical context for the prediction of gravitationalwaves). Notably, learning systems 100 and/or 200 may filter outknowledge items that are not specifically associated with the history ofgravitational waves, such the cause of the gravitational waves and thedetection of gravitational waves, among other possible items.

At step 358, learning systems 100 and/or 200 may generate a set ofstructured learning assets that correspond to the set of filteredknowledge items. In one example, learning systems 100 and/or 200 maygenerate the structured learning assets based on templates as furtherdescribed herein. For example, learning systems 100 and/or 200 maygenerate a structured learning asset based on a “historical discovery”template that includes the following fields: a link to the digitalsource material, the name of the person who made the discovery, and theyear of the discovery. The fields of the “detection method” template maybe populated using data from the filtered knowledge items, such as videoclip describing the prediction of Gravitational Waves, the name AlbertEinstein, and the year 1916.

At step 360, learning systems 100 and/or 200 may generate one or moreinteractions based on the structured learning assets. In one example, acloud-based personalized learning engine and/or artificial intelligenceengine, such as adaptive engines 106 and/or 226, may generate the one ormore interactions. For example, learning systems 100 and/or 200 maygenerate an interaction that includes a question. “Who first predictedthe existence of Gravitational Waves?” The question may be a multiplechoice question, a matching question, and/or a fill-in-the blankquestion, among other possible questions, formats, and/or modalities.

At step 362, learning systems 100 and/or 200 may generate a personalizedlearning experience based on the one or more interactions describedherein. In one example, learning systems 100 and/or 200 may generate thepersonalized learning experiences with interaction applications 109described herein. Consider the example above where learning systems 100and/or 200 generate an interaction that includes a question. “Who firstpredicted the existence of Gravitational Waves?” In such instances, alearner response may be, “Richard Feynman.” Learning systems 100 and/or200 may generate and transmit another interaction that includes, “No,here's a clip to remind you,” including a clip from the structuredlearning asset with the video of the prediction of Gravitational Waves.

At step 364, learning systems 100 and/or 200 may generate user data thatmay be used at step 356 described herein to filter the set of knowledgeitems. In one example, the user data may include or indicate the learnerhistory and/or learner responses to the one or more interactions of thepersonalized learning experience.

In some instances, the user data may indicate one or more learningobjectives, such as the desire to learn about one or more topics and/ornot learn about one or more other topics. For example, the user data mayindicate. “User desires to learn about the history of modern astronomyand discovery, not astrophysics.”

FIGS. 4A-4B are simplified diagrams of a method 400 for extractingknowledge items from digital content according to some embodiments. Insome examples consistent with FIGS. 1-3C, method 400 may be used toimplement steps 304 a. 334, and/or 354 of processes 300, 330, and/or350, respectively. Consistent with such embodiments, method 400 may beperformed by learning systems 100 and/or 200.

At a process 410, digital content is received. In some examples, thedigital content may include one or more digital files associated withdigital source materials, such as online textbooks, instruction videos,journal articles, and/or the like. In various embodiments, the digitalcontent may be unstructured and/or heterogeneous. For example, thedigital content may include raw text. HTML files. PDF files, multimediacontent (e.g., JPEG, MPG, PNG, etc.), compressed files, and/or the like.In some examples, the digital content may be received over the Internetand/or retrieved from local storage.

At a process 420, a text document is generated based on the digitalcontent. In some examples, generating the text document may includeextracting raw text, metadata, and/or other features of the digitalcontent that are representable in a text-based format. In general, thetechniques used to generate the text document may vary based on the typeand/or characteristics of the digital content. For example, when thedigital content includes images of text (e.g., when the digital contentincludes a PDF file and/or scanned text), the text document may begenerated by performing optical character recognition (OCR) on thedigital content to convert the images to text. In another example, whenthe digital content includes text in a markup language (e.g., when thedigital content includes an HTML and/or XML file), the text document maybe generated by parsing and/or “scraping” the digital content to extractrelevant text. In some examples, when the digital content includesmultimedia content, generating the text document may include identifyingmetadata associated with the multimedia content. For example, themetadata may be extracted from the multimedia file itself (e.g., anauthor, title, and/or description of the multimedia work embedded in themultimedia file) and/or retrieved from an external resource (e.g., bylooking up song lyrics from an online database). In some embodiments,the text document may be generated using machine learning techniques.For example, an image recognition machine learning model may be used tolabel objects appearing in an image or video, a speech recognitionmachine learning model may be used to transcribe spoken-word audio, anartificial intelligence model may be used to automatically summarize thedigital content in plain language, and/or the like.

Although numerous embodiments of process 420 are contemplated, FIG. 4Bdepicts a particular embodiment of process 420 that may be used toimprove the quality and/or accuracy of the generated text document incertain circumstances. In some examples, the embodiment of process 420depicted in FIG. 4B may be particularly suited for converting imagesand/or speech to text (e.g., when the digital content includes PDF filesand/or audio files).

Various techniques exist for converting images and/or speech to text,and in general, each technique has different strengths and weaknessesthat may affect the quality of the resulting text document. While aparticular technique may provide the most accurate results some cases,another technique may be more accurate in other cases. Accordingly, theembodiment of process 420 depicted in FIG. 4B may be used to harness thestrengths of various conversion techniques when generating a textdocument from digital content.

At a process 422, a plurality of intermediate text documents aregenerated from the digital content using a corresponding plurality ofconversion techniques. For example, when the digital content includes aPDF file, process 422 may include using a plurality of PDF-to-textconverters to generate the corresponding plurality of intermediate textdocuments. Similarly, when the digital content includes an audio file,process 422 may include using a plurality of speech-to-text converters.Each of the plurality of intermediate text documents may include variousdefects (e.g., OCR errors, speech detection errors, etc.) and/or mayotherwise vary in quality based on the relative strengths and/orweaknesses of the corresponding conversion technique.

At a process 424, a master text document is selected from the pluralityof intermediate text documents. In some examples, the master textdocument may correspond to the highest quality text document among theplurality of intermediate text documents. In some examples, the mastertext document may be selected manually and/or automatically, usingobjective and/or subjective criteria.

Although the master text document may be the highest quality documentamong the plurality of intermediate text documents, the master documentmay still have one or more defects. Accordingly, at a process 426, themaster text document is repaired. In some examples, repairing the mastertext document may include automatically and/or manually identifying andcorrecting defects in the master text document. For example, defects inthe master text document may be patched using corresponding,non-defective portions of one or more of the other intermediate textdocuments. When the master text document has been repaired, method 400may proceed to a process 430 for further processing of the textdocument.

Returning to FIG. 4A, at a process 430, the text document is tokenized.That is, the text document is split into tokens (e.g., paragraphs,sentences, clauses, phrases, words, and/or the like). In some examples,the text document may be tokenized based on white space (e.g., spaces,tabs, return carriages, and/or the like), punctuation (e.g., periods,question marks, exclamation points, and/or the like), grammaticalfeatures (e.g., transition words in a compound sentence), and/or thelike. In some embodiments, natural language processing techniques may beused to accurately identify tokens. For example, natural languageprocessing techniques may be used to distinguish a period marking theend of a sentence from other uses of the period character, such as adecimal point and/or an abbreviation. In some embodiments, process 430may further include determining supplemental information correspondingto the tokens. For example, process 430 may include performinglemmatization to identify a lemma associated with a word token, indexingto identify a dictionary index associated with a word token, and/or thelike. Consistent with such embodiments, the tokenized text documentgenerated at process 430 may include one or more data structures thatcollectively provide access to the raw text of the text document, thetokens appearing in the text document, and/or the supplementalinformation associated with the tokens.

At a process 440, concepts in the tokenized text document areidentified. In some embodiments, the concepts may correspond to nounphrases (i.e., a group of one or more words describing a particularperson, place, or thing). Consistent with such embodiments, the conceptsmay be identified by performing part of speech (POS) tagging and nounchunking based on the tokenized text document. Part of speech taggingincludes identifying the part of speech (e.g., noun, verb, adjective,etc.) of each word in the text document. Chunking includes identifyinggroups of words that form noun phrases based on patterns in thetokenized and tagged text document. In some examples, part of speechtagging and/or noun chunking may be performed using machine learningand/or natural language processing techniques. In some embodiments,process 440 may further include identifying relationships among theconcepts. For example, process 440 may include building a syntacticgraph associated with the tokenized text document, in which theidentified concepts form nodes and the identified relationships amongconcepts form vertices. More broadly, process 440 may includeidentifying and/or describing structural and/or syntactic features ofthe tokenized text document in preparation for modeling and/orunderstanding its content.

At a process 450, a semantic model of the digital content is generated.The semantic model identifies one or more topics and/or sub-topicscovered by the digital content. In some examples, the semantic model maybe identified by performing semantic analysis based on the concepts inthe tokenized text document. In some examples, the semantic model may begenerated using an unsupervised machine learning algorithm. Examples ofunsupervised machine learning algorithms include explicit semanticanalysis (ESA), latent semantic analysis (LSA), and/or the like. In someexamples, the unsupervised machine learning algorithm may performclustering based on the distribution of words in the tokenized textdocument. Examples of clustering techniques include hierarchical clusteranalysis, k-means clustering, Gaussian mixture model, and/or the like.For example, the unsupervised machine learning algorithm may includemodeling the distribution of words in the tokenized text document as amixture of categorical distributions, where each categoricaldistribution corresponds to a topic, and applying a Gaussian mixturemodel to the categorical distributions identify one or more topics inthe tokenized text document. In this manner, a semantic model of thedigital content is formed, where the semantic model may include anoutline of the topics and/or sub-topics covered by the digital content,locations within the digital content that embody the topics and/orsub-topics, and/or the like.

At a process 460, concept scores are assigned to the concepts identifiedat process 440. The concept scores indicate how relevant and/orimportant each concept is to the ideas presented in the digital contentand/or to the user's learning objectives. In some examples, the conceptscore may include a numerical score, a ranking of concepts, aclassification of concepts (e.g., “high.” “medium.” and “low”relevance), a selection of the n most relevant concepts, and/or thelike. In some examples, a given concept score may be assigned based on arelationship between a concept and the semantic model generated atprocess 450. For example, the relationship may be defined by acentrality metric that indicates how central the concept is to thetopics and/or sub-topics in the semantic model. By using the centralitymetric to determine the concept score, a concept that is closely relatedto one or more topics and/or sub-topics in the semantic model mayreceive a higher concept score than a concept that is distinct from thetopics and/or sub-topics in the semantic model. Additionally oralternately, a given concept score may be assigned based on thespecificity of the concept. Accordingly, a concept that is expressedusing generic terms (e.g., “person”) may receive a lower concept scorethan a concept expressed using terms with narrower meanings (e.g.,“doctor” or “professor”). In further examples, a concept score may beassigned based on a relationship between a concept and a user's learningobjectives. For example, if a user indicates that they desire to learnabout a particular topic (e.g., “astronomy”), a concept closely relatedto this topic may receive a higher concept score than an unrelatedand/or tangentially related concept.

At a process 470, a passage score is assigned to each passage of thetokenized text document. For example, a passage may correspond to aphrase, sentence, paragraph, and/or the like. In general, a passagescore indicates whether the passage is likely to include substantiveinformation (e.g., facts, figures, etc.) that pertains to one or moretopics covered by the digital content. In some examples, a passage scoremay include a numerical score, a ranking of passages, a classificationof passages (e.g., “high,” “medium.” and “low” likelihood of containingrelevant information), a selection of the top n passages, and/or thelike.

In some embodiments, each passage score may be determined based on arelationship between the corresponding passage and the semantic model.For example, the passage score may be determined based on concept scoresassigned to one or more concepts that appear in the passage, where eachconcept score is assigned based on a relationship between a concept andthe semantic model as discussed previously with reference to process460. In some examples, the passage score may be determined byaggregating the concept scores of each concept in the passage (e.g., bycomputing the sum, average, weighted average, and/or the like of eachconcept appearing the passage). In some examples, various otherparameters associated with the passage, such as the length of thepassage, the number of concepts in the passage, grammatical features ofthe passage, and/or the like may factor into the passage score. In someexamples, a formula and/or algorithm used to generate the passage scorebased on the various parameters may be updated over time based on userfeedback (e.g., user feedback collected at step 324 of process 300).Such updates may occur manually and/or automatically (e.g., using amachine learning algorithm).

At a process 480, candidate knowledge items are selected from among thepassages of the tokenized text document based on the passage scores. Forexample, selecting the candidate knowledge items may include iteratingthrough the passages in the tokenized text document to determine whethera particular passage qualifies as a candidate knowledge item. Consistentwith such embodiments, a passage may be selected as a candidateknowledge item when the corresponding passage score is above a thresholdvalue, when the passage score is among the top n passage scores overall,when the passage score is among the top n passage scores for aparticular topic and/or sub-topic, and/or the like. In some embodiments,the threshold value may vary based on the topic and/or sub-topic.

In some embodiments, the candidate knowledge items may be selectedadaptively. That is, the selection may account for similarities betweena current passage and other passages that were previously selected ascandidate knowledge items. Adaptive selection may reduce and/oreliminate redundancies among the candidate knowledge items caused byselecting multiple passages that provide the same or similarinformation. In some examples, adaptive selection may include reducing apassage score when a passage includes concepts that have already beencovered in another passage that was selected as a candidate knowledgeitem. For example, adaptive selection may include incrementally reducinga concept score each time the corresponding concept appears in apassage. In this manner, whenever the concept appears in a passage,subsequent passages repeating the same concept are less likely to beselected as candidate knowledge items. Similarly, adaptive selection mayinclude incrementally reducing the concept scores assigned to a group ofconcepts pertaining to the same topic and/or sub-topic each time one ofthe group of concepts appears in a passage.

According to some embodiments consistent with FIGS. 1-3C, upon selectingthe candidate knowledge items at process 470, method 400 may proceed tosteps 306 and/or 356 of process 300. In this regard, the candidateknowledge items selected at process 470 may be presented to a userand/or filtered based on user data, as discussed previously.

FIG. 5 illustrates templates 500 for structured learning assets 512 anduser devices 504, 506, 508, and/or 510 in accordance with an embodimentof the disclosure. In one example, one or more user devices 504, 506,508, and/or 510 may take the form of user devices 204, 206, and/or 208.

In one embodiment, structured learning assets 512 may be generated basedtemplates 500. In one example, templates 500 may include associationtemplates, vocabulary templates, passage templates, image and/or videoregion templates, sequence templates, and/or pattern templates, amongother possible templates.

In one embodiment, one or more interactions may be generated based onstructured learning assets 512. The one or more interactions may betransmitted to one or more user devices 504, 506, 508, and/or 510. Inone example, user devices 504, 506, 508, and/or 510 may display and/orotherwise output the interactions based on the respective hardwareinterfaces user devices 504, 506.508, and/or 510, such audio speakers.

FIG. 6A illustrates a learning system 600 a with a natural userinterface in a cloud-based configuration in accordance with anembodiment of the disclosure. Learning system 600 a allows a user, suchas a learner and/or an instructor, to interact naturally (e.g.,conversationally) with learning system 600 a. For example, learningsystem 600 a may provide the user with one or more of a voice interface,a gesture interface, a chat interface, a drawing interface, and/or thelike. Accordingly, the user may engage with learning system 600 a usingconversational speech, body movements (e.g., pointing, facialexpressions), sketches, and/or the like. These modes of engagement mayenhance the user's learning/teaching experience by offering an intuitiveway to interact with learning system 600 a.

Learning system 600 a includes one or more learner devices 611-619, anatural user interface engine 620, a natural interaction engine 630, andan adaptive engine 640. In some examples, one or more components oflearning system 600 may be communicatively coupled via a network 650,such as an Internet network and/or a local area network.

Learner devices 611-619 each include one or more input/output modules tointeract with the user of learning system 600. Learner devices 611-619may include virtually any device that supports natural user interactionsand is capable of communication over network 650. For example, learnerdevices 611-619 may include an audio input module such as a microphone,a physical input module such as a camera, a touch screen, and/or adrawing pad, a symbolic input module such as a keyboard, and/or thelike. Likewise, learner devices 611-619 may include an audio outputmodule such as a speaker, a physical output module such as an actuatorand/or a haptic device, a visual output module such as a graphicaldisplay, and/or the like. The input/output module(s) may be connected toand/or incorporated into a device capable of communication over network650, such as a watch, phone, tablet, laptop computer, desktop computer,television, gaming console, speaker system, car, virtual/augmentedreality headset. Internet of Things (IoT) object, and/or the like.

Learner devices 611-619 send and receive natural interactions to andfrom natural user interface engine 620 via network 650. A naturalinteraction may include a series of one or more user requests and acorresponding series of one or more responses generated by learningsystem 600 a.

Natural user interface engine 620 receives and processes one or moreuser requests from learner devices 611-619 and determines a user intentassociated with the user request. For example, the user intent mayinclude a request for learning system 600 a to perform a specifiedaction, a query for information, an answer to a question prompt, and/orthe like. In some embodiments, natural user interface engine 620 mayinclude a request handler 622 to determine the user intent. Requesthandler 622 may interpret the user's request using one or more modules.For example, when the user makes a request using conversational speech,request handler 622 may process the user request using a speech-to-textconversion module and/or a natural language processing module. Thesemodules may run locally as part of natural user interface engine 620,and/or may be accessed over network 650 (e.g., using a cloud-basedapplication programming interface (API)).

Natural user interface engine 620 may trigger one or more system eventsbased on the user intent. For example, the system event may includeconverting the user request into a structured representation of the userintent, such as a JSON and/or XML message. In some examples, the systemevent may include sending one or more notification messages thatidentify the user's intent to natural interaction engine 630 via network650. In some examples, the one or more notification messages may includethe structured representation of the user intent.

Natural user interface engine 620 also receives and processes responseinformation from natural interaction engine 630 and synthesizes acorresponding response. For example, the response information mayinclude response dialog, multimedia content, a confirmation that anaction requested by the user has been performed, a prompt for userinput, and/or the like. The synthesized response may include a naturalresponse (e.g., conversational speech), a preformed response (e.g., amultimedia object and/or a scripted message), and/or a combinationthereof. In some embodiments, natural user interface engine 620 mayinclude a response handler 624 to synthesize the response based onreceived response information. Like request handler 622, responsehandler 624 may include one or more modules. For example, responsehandler 624 may include a text-to-speech module that generates a voiceoutput based on a text input.

The synthesized response may be output using the same mode of engagement(e.g., speech, gesture, drawing, and/or text message) and/or on the samelearning device as a received user request. Alternately, oradditionally, the synthesized response may be output using a differentmode of engagement and/or on a different learning device than thereceived user request. For example, the user request may include a voiceinteraction received via a microphone of learning device 611, and thesynthesized response may include (a) a voice interaction output via aspeaker of learning device 611 and/or (b) multimedia content played viaa graphical display of learning device 612.

Natural interaction engine 630 coordinates one or more naturalinteractions between the user and learning system 600 a. In someembodiments, natural interaction engine 630 may include one or morenatural interaction handlers 631-639 that each correspond to apreconfigured natural interaction. In some examples, one or more ofnatural interaction handlers 631-639 may be invoked based on theparticular user intent determined by natural user interface engine 620.For example, each of natural interaction handlers 631-639 may generateresponse information that is responsive to a different type of userrequest.

In some examples, natural interaction engine 630 may send and/or receiveinformation from adaptive engine 640 via network 650. Adaptive engine640 generally corresponds to adaptive engines 106 and/or 226, asdiscussed previously with respect to FIGS. 1-5. Consistent with suchembodiments, adaptive engine 610 may retrieve knowledge items, such asknowledge items 116, knowledge items 220-224, and/or structured learningassets 512, from a knowledge item bank, such as knowledge item bank 104.In some examples, adaptive engine 610 may generate or obtaininteractions, such as interactions 118 and/or 230, based on theretrieved knowledge items. Accordingly, to generate response informationin response to a user request, natural interaction engine 630 maycommunicate with adaptive engine 640 to retrieve one or more knowledgeitems and/or interactions from adaptive engine 640.

By way of example, learning system 600 a may be implemented using acloud-based virtual assistant with voice interaction capabilities. Inthis example, at least one of learner devices 611-619 may be implementedusing an Internet-enabled speaker connected via network 650 to naturaluser interface engine 620, which implements the cloud-based virtualassistant. A learner makes a user request through the Internet-enabledspeaker by speaking conversationally. The Internet-enabled speaker sendsa digital representation of the user request to the cloud-based virtualassistant over network 650. The cloud-based virtual assistant performsspeech-to-text conversion and natural language processing on the userrequest to determine the user intent. The cloud-based virtual assistantdetermines whether the user intent matches one of natural interactionhandlers 631-639 provided by natural interaction processor 630. Whenthere is a match, the cloud-based virtual assistant sends a message tonatural interaction processor 630 with information associated with theuser intent. The matching natural interaction handler generates aresponse dialog to the user request and sends a response dialog to thecloud-based virtual assistant. The cloud-based virtual assistantconverts the response dialog to a voice response and sends a digitalrepresentation of the voice response to the Internet-enabled speaker,which plays the voice response to the learner. In some examples, theresponse dialog may further include a multimedia component thatinstructs the cloud-based virtual assistant to render multimedia contentto the learner via a second one of learner devices 611-619 that has agraphical display.

FIG. 6B illustrates a learning system 600 b with a natural userinterface in a device-based configuration in accordance with anembodiment of the disclosure. Like learning system 600 a, learningsystem 600 b generally includes the same components as learning system600 a for providing natural interactions with a user. However, unlikethe cloud-based configuration depicted in FIG. 6A, learning system 600 bis arranged in a device-based configuration with natural user interfaceengine 620 and natural interaction engine 630 being implemented locallyon a learner device 610. For example, natural user interface engine 620and/or natural interaction engine 630 may be implemented using one ormore dedicated hardware processors (e.g., dedicated chips) of learnerdevice 610. In some example, learner device 610 may be a modular device,in which case the one or more dedicated hardware processors may beattachable/removable from learner device 610. In another example,natural user interface engine 620 and/or natural interaction 630 may beimplemented using software instructions stored in a memory of learnerdevice 610 and executed on one or more processors of learner device 610.In some examples, natural user interface engine 620 and/or naturalinteraction engine 630 may be run as virtual machines on learner device610.

Although learning systems 600 a and 600 b have been depicted in acloud-based and device-based configuration, respectively, it is to beunderstood that various other arrangements are possible. For example, ina hybrid configuration, a portion of natural user interface engine 620and/or natural interaction engine 630 may be implemented locally onlearner device 610, and another portion may be accessed via network 650.

FIG. 7 illustrates a method 700 for providing a natural interaction witha learning system according to some embodiments. According to someembodiments consistent with FIGS. 6A and 6B, method 700 may be performedby a natural interaction provider, such as natural interaction engine630 of learning system 600 a and/or 600 b.

At an optional process 710, a natural user interface account is linkedwith a personal learning account. The natural user interface account maycorrespond to an account of a user of a learning device, such aslearning devices 610-619, with a natural user interface provider, suchas natural user interface engine 620. The personal learning account maycorrespond to an account of the user with an adaptive engine, such asadaptive engine 630. In order to allow the user to access the adaptiveengine through the natural user interface provider, the natural userinterface account and the personal learning account may be linked. Forexample, the user may provide credentials to the natural user interfaceprovider to grant the natural user interface provider access to thepersonal learning account. In this manner, the natural interactionprovider is able to generate responses to user requests that incorporatecredential-protected information associated with the personal learningaccount of the user.

At a process 720, a notification that identifies a user intentassociated with a user request is received from the natural userinterface provider. In some examples, the notification is received overa network, such as network 650. In some examples, the notification maybe formatted as a JSON and/or XML message. Receipt of the notificationmessage may trigger one or more natural interaction handlers, such asnatural interaction handlers 631-639, to process and respond to thenotification message, as discussed below with respect to processes730-760.

At a process 730, it is determined whether the user intent includes arequest for information. When the user intent includes a request forinformation, method 700 proceeds to a process 735, in which aninformational response is generated and sent to the user via the naturaluser interface provider. For example, a user may say, “System, can youaccess my account?” The natural interaction provider may provide aninformative response that provides, “You last reviewed one day ago andyou have thirty-nine fading memories. Use this mobile phone to refreshyour fading memories.” In another example, the user may say. “Can I havea study tip?” The natural interaction provider may generate aninformative response that provides. “Don't cram. Little and often is therecipe for success.” The user may say, “How is my Napa Valley set?” Thenatural interaction provider may generate an informative response thatprovides, “You are at level 4.1 with a goal of 5.0. You have 3 fadingmemories. It's time to refresh.” In one or more of the precedingexamples, the natural interaction provider may obtain the informationthat is included in the informative response from the adaptive enginevia the network.

At a process 740, it is determined whether the user intent includes arequest to perform an action. When the user intent includes a request toperform an action, method 700 proceeds to a process 745, in which anaction-oriented response is generated and sent to the user via thenatural user interface provider. The action-oriented response mayinclude a confirmation that an action has been and/or will be performed,a prompt for user input, and/or the like. The action-oriented responsemay further include information relevant the requested action. Forexample, the user may say, “Can I do a trial now?” The naturalinteraction provider may generate an action-oriented response thatprovides. “Space exploration set is at level 0.8 with a goal of 2.0. Youhave four fading memories. Would you like to get started?” The user mayrespond, “Yes.” In some examples, the natural interaction provider maytransmit a notification of the requested action to the adaptive enginevia the network.

At a process 750, it is determined whether the user intent includes arequest for a trial. When the user intent includes a request for atrial, method 700 proceeds to a process 755, in which the naturalinteraction provider initiates a question-answer-explanation dialogsequence. The question-answer-explanation dialog sequence may begenerated by retrieving one or more knowledge items and/or interactionsprovided by the adaptive engine. In an exemplaryquestion-answer-explanation dialog sequence, the natural interactionprovider may generate and send a question to the user. When the userresponds with an answer attempt, the natural interaction provider maydetermine whether the answer attempt matches a correct answer to thequestion. The natural interaction provider may then provide anoutcome-based explanation based on whether the user's answer attemptmatches the correct answer. When the user's answer is correct, thenatural interaction provider may generate and send response thatcongratulates the user and/or otherwise inform the user that the answeris correct. When the user's answer is fully or partially incorrect, thenatural interface provider may generate and send a response that informsthe user that the answer is incorrect, informs the user of the correctanswer, informs the user how close or far the user's answer attempt isfrom the correct answer, guides the user to the correct answer, and/orthe like. In some examples, the response may include additionalexplanatory information to reinforce the subject matter embodied in thequestion, such as an explanation of why the user's answer is correct orincorrect.

For example, the natural interaction provider may pose a question thatprovides, “What is the speed of light in kilometers per second?” Theuser may answer, “I think it's 290,000.” In this example, the naturalinteraction provider may extract the answer attempt of “290.000” fromthe voice response of “I think it's 290,000.” In some examples, thenatural interaction provider may generate an explanatory response thatprovides, “That's very close. It's actually 300,000 kilometers persecond.” Notably, if the user responded. “186,000 miles per second.” theexplanatory response may include information that guides the user to acorrect answer, e.g., “Can you provide that in kilometers per second?”

In the example above, the natural interaction provider performs the stepof extracting the numerical value “290.000” from the sentence “I thinkit's 290,000.” However, in some examples, the natural interactionprovider may provide an answer template (e.g., answer slots) to thenatural user interface provider for each question. The answer templateindicates the expected or required format of the user's response, suchas a numerical value, a multiple choice selection (e.g., “true,”“false,” “yes,” “no.” “A,” “B.” “C,” or “D”), and/or the like. In thismanner, the natural user interface provider, rather than the naturalinteraction provider, is responsible for extracting the user's answerfrom a particular user utterance and transmitting the answer to thenatural interaction provider in the expected format.

In one example, at the end of a particular question-answer-explanationdialog sequence, the natural interaction provider may initiate one ormore additional question-answer-explanation dialog sequences based onthe outcome of the first trial and/or the availability of additionalknowledge items and/or interactions from the adaptive engine. In oneexample, the natural interaction provider may generate and send a secondquestion to the user: “Can you order the following planets from smallestto largest: Saturn. Earth. Jupiter. Mars?” The natural interactionprovider may receive an answer attempt from the user via the naturaluser interface provider: “Earth, Mars, Jupiter, Saturn.” In one example,the natural interaction provider may generate and transmit a correctionthat provides, “No, the correct order from smallest to largest is Mars,Earth, Saturn, Jupiter. Jupiter has a diameter 20 times larger than thatof Mars.”

At a process 760, it is determined whether the user intent includes arequest to learn a knowledge item. When the user intent includes arequest to learn a knowledge item, method 700 proceeds to a process 765,in which a teaching response is generated and sent to the user via thenatural user interface provider. In some examples, requested knowledgeitem may correspond to knowledge item 118 and or structured learningassets 512. In one example, the teaching response may include a one ormore of a natural response, such as conversational speech, and/or apreformed multimedia interaction (e.g., text, audio, video, and/or thelike). For example, the teaching response may provide, “Let me show youa video of a leading expert in astronomy [the video may be played on theuser device].” The video may provide, “ . . . the length of the meter isdefined from the speed of light and the international standard for time. . . ”

In one example, the teaching response may include a follow-up evaluationto determine how well the user learned the knowledge item. In someexamples, the follow-up evaluation may be in the format of aquestion-answer-explanation dialog sequence. For example, after showingthe user the video of the astronomy expert, the natural interactionprovider may pose a question to the user that provides, “Do you knowwhat base unit of length is defined by the speed of light and theinternational standard of time?” The user may respond, “the meter.” Thenatural interaction provider may respond. “Yes, that is correct.”

Where applicable, various embodiments provided by the present disclosurecan be implemented using hardware, software, or combinations of hardwareand software. Also where applicable, the various hardware componentsand/or software components set forth herein can be combined intocomposite components comprising software, hardware, and/or both withoutdeparting from the spirit of the present disclosure. Where applicable,the various hardware components and/or software components set forthherein can be separated into sub-components comprising software,hardware, or both without departing from the spirit of the presentdisclosure. In addition, where applicable, it is contemplated thatsoftware components can be implemented as hardware components, andvice-versa.

Software in accordance with the present disclosure, such asnon-transitory instructions, program code, and/or data, can be stored onone or more non-transitory machine readable mediums. It is alsocontemplated that software identified herein can be implemented usingone or more general purpose or specific purpose computers and/orcomputer systems, networked and/or otherwise. Where applicable, theordering of various steps described herein can be changed, combined intocomposite steps, and/or separated into sub-steps to provide featuresdescribed herein.

Embodiments described above illustrate but do not limit the invention.It should also be understood that numerous modifications and variationsare possible in accordance with the principles of the invention.Accordingly, the scope of the invention is defined only by the followingclaims.

What is claimed is:
 1. A learning system comprising: a non-transitorymemory; and one or more hardware processors configured to readinstructions from the non-transitory memory to cause the system toperform operations comprising: generating a semantic model based on asource material using a machine learning algorithm; assigning aplurality of concept scores to a corresponding plurality of conceptsincluded in the source material, wherein each of the plurality ofconcept scores is assigned based on a relationship between a respectiveone of the plurality of concepts and the semantic model; selecting oneor more candidate knowledge items from the source material based on theplurality of concept scores; generating an interaction based at least onthe one or more candidate knowledge items; and transmitting theinteraction to a user device.
 2. The learning system of claim 1, whereinthe operations further comprise: filtering the one or more candidateknowledge items based on user data to provide one or more filteredcandidate knowledge items; and generating the interaction based at leaston the one or more filtered candidate knowledge items.
 3. The learningsystem of claim 2, wherein the filtering of the one or more candidateknowledge items based on the user data includes identifying one or morelearning objectives of a user of the user device; and the one or morelearning objectives of the user correspond to a query provided by theuser of the user device.
 4. The learning system of claim 2, wherein thefiltering of the one or more candidate knowledge items based on the userdata includes identifying one or more learning objectives of a user ofthe user device; and the one or more learning objectives are generatedautomatically based on existing knowledge of the user.
 5. The learningsystem of claim 4, wherein the one or more learning objectives areautomatically generated by comparing the existing knowledge of the userof the one or more candidate knowledge items with a predeterminedthreshold; when the existing knowledge of the user of the one or morecandidate knowledge items is greater than the predetermined threshold itis determined that re-learning of the one or more candidate knowledgeitems is not necessary.
 6. The learning system of claim 2, wherein thefiltering of the one or more candidate knowledge items based on the userdata includes identifying one or more learning objectives of a user ofthe user device; and the one or more candidate knowledge items arefiltered by determining a relevance score for each of the one or morecandidate knowledge items and determining whether the relevance scoremeets a threshold.
 7. The learning system of claim 2, wherein theoperations further comprise: generating user data that includes one ormore metrics that indicate a level of retention or understanding of theone or more candidate knowledge items by a user of the user device; anddetermining the one or more metrics based on the user's responses to oneor more previous interactions.
 8. The learning system of claim 7,wherein the one or more metrics are further determined based onhistorical user responses associated with other knowledge items orinteractions.
 9. The learning system of claim 1, wherein the sourcematerial includes at least one of a text, a video, an image, or audiocontent.
 10. The learning system of claim 1, wherein the interactionincludes a voice interaction.
 11. The learning system of claim 1,wherein the one or more candidate knowledge items are selectedadaptively to reduce redundancies among the one or more candidateknowledge items.
 12. The learning system of claim 1, wherein each of theplurality of concept scores indicates a level of relevance between arespective concept included in the source material and one or morelearning objectives of a user of the user device; and the plurality ofconcepts included in the source material are ranked based on theplurality of concept scores.
 13. The learning system of claim 1, whereinthe relationship is defined by a centrality metric that indicates howcentral a corresponding concept is to one or more topics in the semanticmodel.
 14. The learning system of claim 1, wherein the interactiontransmitted to the user device is an interaction of a course; the courseis completed by a user of the user device in order to obtain aprofessional certification.
 15. The learning system of claim 1, whereinthe operations further comprise: sending, to the user device, analyticsdata that indicate a performance result based on a respective responseto the interaction by a user of the user device.
 16. The learning systemof claim 15, wherein the operations further comprise: sending theanalytics data to a device of an individual other than the user of theuser device to inform the individual whether or not the user understandsat least one of the one or more candidate knowledge items based on whichthe interaction was generated.
 17. A method comprising: generating asemantic model based on a source material using a machine learningalgorithm; assigning a plurality of concept scores to a correspondingplurality of concepts included in the source material, wherein each ofthe plurality of concept scores is assigned based on a relationshipbetween a respective one of the plurality of concepts and the semanticmodel; selecting one or more candidate knowledge items from the sourcematerial based on the plurality of concept scores; generating aninteraction based at least on the one or more candidate knowledge items;and transmitting the interaction to a user device.
 18. The method ofclaim 17, further comprising: filtering the one or more candidateknowledge items based on user data to provide one or more filteredcandidate knowledge items; and generating the interaction based at leaston the one or more filtered candidate knowledge items.
 19. Anon-transitory machine-readable medium having stored thereonmachine-readable instructions executable to cause a machine to performoperations comprising: generating a semantic model based on a sourcematerial using a machine learning algorithm; assigning a plurality ofconcept scores to a corresponding plurality of concepts included in thesource material, wherein each of the plurality of concept scores isassigned based on a relationship between a respective one of theplurality of concepts and the semantic model; selecting one or morecandidate knowledge items from the source material based on theplurality of concept scores; generating an interaction based at least onthe one or more candidate knowledge items; and transmitting theinteraction to a user device.
 20. The non-transitory machine-readablemedium of claim 19, wherein generating the interaction includes:filtering the one or more candidate knowledge items based on user datato provide one or more filtered candidate knowledge items; andgenerating the interaction based at least on the one or more filteredcandidate knowledge items.